Efficient Neural Architecture Search Research Articles

A Training-Free Neural Architecture Search Algorithm Based on Search Economics

Motivated by the observation that most neural architecture search (NAS) methods are time-consuming because a “training process” is required to evaluate each searched neural architecture, this paper presents an efficient NAS algorithm based on a promising metaheuristic algorithm named search economics (SE) and a new training-free estimator to evaluate the searched neural architectures for not only obtaining a good neural architecture but also accelerating the computation time. The basic idea of the proposed NAS algorithm is to use the so-called expected value of each region in the search space to guide the search so that it will focus on searching high potential regions instead of solutions with high objective values in particular regions. To evaluate the performance of the proposed algorithm, we compare it with state-of-the-art non-training-free and training-free NAS methods. Experimental results show that the proposed algorithm is capable of finding a result that is similar to or better than those found by most non-training-free NAS algorithms compared in this study but taking only a tiny portion of the computation time.

Evolution and Efficiency in Neural Architecture Search: Bridging the Gap between Expert Design and Automated Optimization

The paper provides a comprehensive overview of Neural Architecture Search (NAS), emphasizing its evolution from manual design to automated, computationally driven approaches. It covers the inception and growth of NAS, highlighting its application across various domains, including medical imaging and natural language processing. The document details the shift from expert-driven design to algorithm-driven processes, exploring initial methodologies like reinforcement learning and evolutionary algorithms. It also discusses the challenges of computational demands and the emergence of efficient NAS methodologies, such as Differentiable Architecture Search and hardware-aware NAS. The paper further elaborates on NAS's application in computer vision, NLP, and beyond, demonstrating its versatility and potential for optimizing neural network architectures across different tasks. Future directions and challenges, including computational efficiency and the integration with emerging AI domains, are addressed, showcasing NAS's dynamic nature and its continued evolution towards more sophisticated and efficient architecture search methods.

ENAS-B: Combining ENAS With Bayesian Optimization for Automatic Design of Optimal CNN Architectures for Breast Lesion Classification From Ultrasound Images.

Efficient Neural Architecture Search (ENAS) is a recent development in searching for optimal cell structures for Convolutional Neural Network (CNN) design. It has been successfully used in various applications including ultrasound image classification for breast lesions. However, the existing ENAS approach only optimizes cell structures rather than the whole CNN architecture nor its trainable hyperparameters. This paper presents a novel framework for automatic design of CNN architectures by combining strengths of ENAS and Bayesian Optimization in two-folds. Firstly, we use ENAS to search for optimal normal and reduction cells. Secondly, with the optimal cells and a suitable hyperparameter search space, we adopt Bayesian Optimization to find the optimal depth of the network and optimal configuration of the trainable hyperparameters. To test the validity of the proposed framework, a dataset of 1522 breast lesion ultrasound images is used for the searching and modeling. We then evaluate the robustness of the proposed approach by testing the optimized CNN model on three external datasets consisting of 727 benign and 506 malignant lesion images. We further compare the CNN model with the default ENAS-based CNN model, and then with CNN models based on the state-of-the-art architectures. The results (error rate of no more than 20.6% on internal tests and 17.3% on average of external tests) show that the proposed framework generates robust and light CNN models.

EGFA-NAS: a neural architecture search method based on explosion gravitation field algorithm

Neural architecture search (NAS) is an extremely complex optimization task. Recently, population-based optimization algorithms, such as evolutionary algorithm, have been adopted as search strategies for designing neural networks automatically. Various population-based NAS methods are promising in searching for high-performance neural architectures. The explosion gravitation field algorithm (EGFA) inspired by the formation process of planets is a novel population-based optimization algorithm with excellent global optimization capability and remarkable efficiency, compared with the classical population-based algorithms, such as GA and PSO. Thus, this paper attempts to develop a more efficient NAS method, called EGFA-NAS, by utilizing the work mechanisms of EGFA, which relaxes the search discrete space to a continuous one and then utilizes EGFA and gradient descent to optimize the weights of the candidate architectures in conjunction. To reduce the computational cost, a training strategy by utilizing the population mechanism of EGFA-NAS is proposed. In addition, a weight inheritance strategy for the new generated dust individuals is proposed during the explosion operation to improve performance and efficiency. The performance of EGFA-NAS is investigated in two typical micro search spaces: NAS-Bench-201 and DARTS, and compared with various kinds of state-of-the-art NAS competitors. The experimental results demonstrate that EGFA-NAS is able to match or outperform the state-of-the-art NAS methods on image classification tasks with remarkable efficiency improvement.

Improving Differentiable Architecture Search via self-distillation

Differentiable Architecture Search (DARTS) is a simple yet efficient Neural Architecture Search (NAS) method. During the search stage, DARTS trains a supernet by jointly optimizing architecture parameters and network parameters. During the evaluation stage, DARTS discretizes the supernet to derive the optimal architecture based on architecture parameters. However, recent research has shown that during the training process, the supernet tends to converge towards sharp minima rather than flat minima. This is evidenced by the higher sharpness of the loss landscape of the supernet, which ultimately leads to a performance gap between the supernet and the optimal architecture. In this paper, we propose Self-Distillation Differentiable Neural Architecture Search (SD-DARTS) to alleviate the discretization gap. We utilize self-distillation to distill knowledge from previous steps of the supernet to guide its training in the current step, effectively reducing the sharpness of the supernet’s loss and bridging the performance gap between the supernet and the optimal architecture. Furthermore, we introduce the concept of voting teachers, where multiple previous supernets are selected as teachers, and their output probabilities are aggregated through voting to obtain the final teacher prediction. Experimental results on real datasets demonstrate the advantages of our novel self-distillation-based NAS method compared to state-of-the-art alternatives.

Efficient neural architecture search for emotion recognition

Efficient neural architecture search for emotion recognition

EST-NAS: An evolutionary strategy with gradient descent for neural architecture search

Using weight-sharing and continuous relaxation strategies, the gradient descent-based differential architecture search has achieved great success in automatically designing neural network architectures. However, unresolved issues, i.e., the local optimum dilemma of the gradient descent method, and the network performance collapse of the searched architecture with too many unreasonable operations, are still frustrating for researchers and practitioners. To address these two issues, a novel and efficient neural architecture search approach based on a hybrid evolutionary strategy, termed EST-NAS, is proposed in this paper. In particular, we propose using a new evolutionary strategy to explore various search directions based on the gradient descent-based neural network architecture search, aiming at obtaining a more excellent architecture. In the proposed EST-NAS, the gradient descent architecture search is performed first, and then the best architecture obtained is utilized to design an efficient initialization for the following evolutionary strategy-based architecture search. By hybridizing evolutionary strategy with gradient descent-based search, EST-NAS can improve the performance of the searched architecture with better search efficiency. Meanwhile, the validation accuracy is applied to directly measure the importance of operations, which reduces the error in the relationship between operation and task performance. Extensive experimental results in the various datasets on different search spaces show that the proposed EST-NAS achieves remarkably competitive performance with less search cost, compared to other state-of-the-art NAS approaches.

An efficient neural architecture search based deep learning algorithm for surface defects detection of T-beam

An efficient neural architecture search based deep learning algorithm for surface defects detection of T-beam

An efficient neural architecture search based deep learning algorithm for surface defects detection of T-beam

An efficient neural architecture search based deep learning algorithm for surface defects detection of T-beam

Toward efficient neural architecture search with dynamic mapping-adaptive sampling for resource-limited edge device

Toward efficient neural architecture search with dynamic mapping-adaptive sampling for resource-limited edge device

Neural architecture search using property guided synthesis

Neural architecture search (NAS) has become an increasingly important tool within the deep learning community in recent years, yielding many practical advancements in the design of deep neural network architectures. However, most existing approaches operate within highly structured design spaces, and hence (1) explore only a small fraction of the full search space of neural architectures while also (2) requiring significant manual effort from domain experts. In this work, we develop techniques that enable efficient NAS in a significantly larger design space. In particular, we propose to perform NAS in an abstract search space of program properties. Our key insights are as follows: (1) an abstract search space can be significantly smaller than the original search space, and (2) architectures with similar program properties should also have similar performance; thus, we can search more efficiently in the abstract search space. To enable this approach, we also introduce a novel efficient synthesis procedure, which performs the role of concretizing a set of promising program properties into a satisfying neural architecture. We implement our approach, αNAS, within an evolutionary framework, where the mutations are guided by the program properties. Starting with a ResNet-34 model, αNAS produces a model with slightly improved accuracy on CIFAR-10 but 96% fewer parameters. On ImageNet, αNAS is able to improve over Vision Transformer (30% fewer FLOPS and parameters), ResNet-50 (23% fewer FLOPS, 14% fewer parameters), and EfficientNet (7% fewer FLOPS and parameters) without any degradation in accuracy.

A Lightweight Deep Learning Framework for Galaxy Morphology Classification

With the construction of large telescopes and the explosive growth of observed galaxy data, we are facing the problem to improve the data processing efficiency while ensuring the accuracy of galaxy morphology classification. Therefore, this work designed a lightweight deep learning framework, EfficientNet-G3, for galaxy morphology classification. The proposed framework is based on EfficientNet which integrates the Efficient Neural Architecture Search algorithm. Its performance is assessed with the data set from the Galaxy Zoo Challenge Project on Kaggle. Compared with several typical neural networks and deep learning frameworks in galaxy morphology classification, the proposed EfficientNet-G3 model improved the classification accuracy from 95.8% to 96.63% with F1-Score values of 97.1%. Typically, this model uses the least number of parameters, which is about one tenth that of DenseNet161 and one fifth that of ResNet-26, but its accuracy is about one percent higher than them. The proposed EfficientNet-G3 can act as an important reference for fast morphological classification for massive galaxy data in terms of efficiency and accuracy.

BNAS: Efficient Neural Architecture Search Using Broad Scalable Architecture.

Efficient neural architecture search (ENAS) achieves novel efficiency for learning architecture with high-performance via parameter sharing and reinforcement learning (RL). In the phase of architecture search, ENAS employs deep scalable architecture as search space whose training process consumes most of the search cost. Moreover, time-consuming model training is proportional to the depth of deep scalable architecture. Through experiments using ENAS on CIFAR-10, we find that layer reduction of scalable architecture is an effective way to accelerate the search process of ENAS but suffers from a prohibitive performance drop in the phase of architecture estimation. In this article, we propose a broad neural architecture search (BNAS) where we elaborately design broad scalable architecture dubbed broad convolutional neural network (BCNN) to solve the above issue. On the one hand, the proposed broad scalable architecture has fast training speed due to its shallow topology. Moreover, we also adopt RL and parameter sharing used in ENAS as the optimization strategy of BNAS. Hence, the proposed approach can achieve higher search efficiency. On the other hand, the broad scalable architecture extracts multi-scale features and enhancement representations, and feeds them into global average pooling (GAP) layer to yield more reasonable and comprehensive representations. Therefore, the performance of broad scalable architecture can be promised. In particular, we also develop two variants for BNAS that modify the topology of BCNN. In order to verify the effectiveness of BNAS, several experiments are performed and experimental results show that 1) BNAS delivers 0.19 days which is 2.37× less expensive than ENAS who ranks the best in RL-based NAS approaches; 2) compared with small-size (0.5 million parameters) and medium-size (1.1 million parameters) models, the architecture learned by BNAS obtains state-of-the-art performance (3.58% and 3.24% test error) on CIFAR-10; and 3) the learned architecture achieves 25.3% top-1 error on ImageNet just using 3.9 million parameters.

NASMDR: a framework for miRNA-drug resistance prediction using efficient neural architecture search and graph isomorphism networks.

As a frontier field of individualized therapy, microRNA (miRNA) pharmacogenomics facilitates the understanding of different individual responses to certain drugs and provides a reasonable reference for clinical treatment. However, the known drug resistance-associated miRNAs are not yet sufficient to support precision medicine. Although existing methods are effective, they all focus on modelling miRNA-drug resistance interaction graphs, making their performance bounded by the interaction density. In this study, we propose a framework for miRNA-drug resistance prediction through efficient neural architecture search and graph isomorphism networks (NASMDR). NASMDR uses attribute information instead of the commonly used interactive graph information. In the cross-validation experiment, the proposed framework can achieve an AUC of 0.9468 on the ncDR dataset, which is 2.29% higher than the state-of-the-art method. In addition, we propose a novel sequence characterization approach, k-mer Sparse Nonnegative Matrix Factorization (KSNMF). The results show that NASMDR provides novel insights for integrating efficient neural architecture search and graph isomorphic networks into a unified framework to predict drug resistance-related miRNAs. The codes for NASMDR are available at https://github.com/kaizheng-academic/NASMDR.

Towards One Shot Search Space Poisoning in Neural Architecture Search (Student Abstract)

We evaluate the robustness of a Neural Architecture Search (NAS) algorithm known as Efficient NAS (ENAS) against data agnostic poisoning attacks on the original search space with carefully designed ineffective operations. We empirically demonstrate how our one shot search space poisoning approach exploits design flaws in the ENAS controller to degrade predictive performance on classification tasks. With just two poisoning operations injected into the search space, we inflate prediction error rates for child networks upto 90% on the CIFAR-10 dataset.

Adaptive feature extraction and fine‐grained modulation recognition of multi‐function radar under small sample conditions

Multi-function radars (MFRs) are sophisticated sensors with fine-grained modes, which modify their modulation types and parameters range generating various signals to fulfil different tasks, such as surveillance and tracking. In electromagnetic reconnaissance, recognition of MFR fine-grained modes can provide a basis for analysing strategies and reaction. With the limit of real applications, it is hard to obtain a large number of labelled samples for existing methods to learn the difference between categories. Therefore, it is essential to develop new methods to extract general knowledge of MFRs and identify modes with only a few samples. This paper proposes a few-shot learning (FSL) framework based on efficient neural architecture search (ENAS) with high robustness and portability, which designs a suitable network structure automated and quickly adapts to new environments. The experimental results show that the proposed method can still achieve excellent fine-grained modulation recognition performance (92.6%) under the condition of -6 dB signal-to-noise ratio (SNR), even when each class only provides one fixed-duration signal sample. The robustness is also verified under different conditions.

TouchNAS: Efficient Touch Detection Model Design Methodology for Resource-Constrained Devices

The multi-touch algorithm in touch screens is one of the most important interaction tasks for functional operations. This algorithm classifies multiple objects and recognizes the touch positions. However, rule-based algorithms have limited detection performance for small and closely-spaced touch objects. Deep learning-based multi-touch algorithms achieve higher accuracy than conventional rule-based methods. However, the implementation of state-of-the-art multi-touch algorithms in hardware faces the critical problem that the latency and necessary hardware resources are greatly increased owing to the significant computational complexity. Neural architecture search (NAS) has recently been recognized as an effective solution to search for optimal deep learning models in hardware. This paper proposes an efficient NAS algorithm, TouchNAS, to find the optimal multi-touch algorithm on a target device. We first introduce the progressive architecture pruning method for the search space. This method reduces the training and evaluation costs by omitting the accuracy predictor. We also propose a dynamic learning rate scheduler that assists the efficient training of multiple models by calculating the number of model parameters. Finally, we present multi-touch datasets to measure the performance of multi-touch algorithms objectively. The experimental results on various deep learning models show that our TouchNAS identifies only the candidates for the optimal model within the entire search space for selection and training. Moreover, the optimal model found by TouchNAS achieves superior performance over state-of-the-art NAS methods.

BNNAS++: Towards Unbiased Neural Architecture Search With Batch Normalization

Neural Architecture Search (NAS) achieves significant progress in many computer vision tasks, yet training and searching high-performance architectures over large search space are time-consuming. The NAS with BatchNorm (BNNAS) is an efficient NAS algorithm to speed up the training of supernet ten times over conventional schemes by fixing the convolutional layers and training the BatchNorm only. In this paper, we systematically examine the limitation of the BNNAS and present improved techniques. We observe that the BNNAS is prone to select convolution and fails to find plausible architectures with small FLOPs, especially on the search space with operations other than convolutions. This is because the number of learnable parameters on BatchNorm is imbalanced for different operations, i.e., BatchNorm is only attached to the convolutional layer. To fix the issue,We present the unbiased BNNAS for training. However, our empirical evidence shows that the performance indicator in BNNAS is ineffective on the unbiased BNNAS. To this end, we propose a novel performance indicator, which decomposes the model performance into model <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">expressivity, trainability</i> , and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">uncertainty</i> , to preserve the rank consistency on the unbiased BNNAS. Our proposed NAS method, named as BNNAS++, is robust on various search spaces and can efficiently search high-performance architectures with small FLOPs. We validate the effectiveness of BNNAS++ on NAS-Bench-101, NAS-Bench-201, DARTS search space, and MobileNet search space, showing superior performance over existing methods.

Bringing AI to edge: From deep learning’s perspective

Edge computing and artificial intelligence (AI), especially deep learning algorithms, are gradually intersecting to build the novel system, namely edge intelligence. However, the development of edge intelligence systems encounters several challenges, and one of these challenges is the computational gap between computation-intensive deep learning algorithms and less-capable edge systems. Due to the computational gap, many edge intelligence systems cannot meet the expected performance requirements. To bridge the gap, a plethora of new techniques and optimization methods were proposed in the past years: lightweight deep learning models, network compression, and efficient neural architecture search. Although some reviews or surveys have partially covered this large body of literature, we lack a systematic and comprehensive review to discuss all aspects of these deep learning techniques which are critical for edge intelligence implementation. As various and diverse methods, applicable to edge systems, are proposed, a holistic review would enable edge computing engineers and the community to understand the state-of-the-art deep learning techniques that are instrumental for edge intelligence and to facilitate the development of edge intelligence systems. This paper surveys the representative and latest deep learning techniques that are useful for edge intelligence systems, including hand-crafted models, model compression, hardware-aware neural architecture search, and adaptive deep learning models. Finally, based on observations and simple experiments we conducted, we discuss some future directions.

AdvantageNAS: Efficient Neural Architecture Search with Credit Assignment

Neural architecture search (NAS) is an approach for automatically designing a neural network architecture without human effort or expert knowledge. However, the high computational cost of NAS limits its use in commercial applications. Two recent NAS paradigms, namely one-shot and sparse propagation, which reduce the time and space complexities, respectively, provide clues for solving this problem. In this paper, we propose a novel search strategy for one-shot and sparse propagation NAS, namely AdvantageNAS, which further reduces the time complexity of NAS by reducing the number of search iterations. AdvantageNAS is a gradient-based approach that improves the search efficiency by introducing credit assignment in gradient estimation for architecture updates. Experiments on the NAS-Bench-201 and PTB dataset show that AdvantageNAS discovers an architecture with higher performance under a limited time budget compared to existing sparse propagation NAS. To further reveal the reliabilities of AdvantageNAS, we investigate it theoretically and find that it monotonically improves the expected loss and thus converges.